Skip to content

Ci/metal migration - #142

Open
orionpapadakis wants to merge 12 commits into
mainfrom
ci/metal-migration
Open

Ci/metal migration#142
orionpapadakis wants to merge 12 commits into
mainfrom
ci/metal-migration

Conversation

@orionpapadakis

Copy link
Copy Markdown
Collaborator

This PR:

  1. Drops PTX backend, and adds Metal on Mac self-hosted runner — removed all PTX matrix entries and PTX-only CUDA-graph steps from build-and-run.yml; added metal as a third backend that runs on [self-hosted, macOS] while opencl/cuda stay on [self-hosted, Linux], using a matrix.backend.os field to route runs-on per entry.
  2. Provides a fix for Metal backend: avoid split-KV attention kernel Qwen3 crashes on Metal backend — root-caused and fixed the Qwen3 Metal crash: TornadoVM's Metal backend fails to JIT the multi-workgroup split-KV flash-decoding attention kernel (processHeadsFlashAttentionSplitKV) — confirmed by tracing crash logs down to kernel submitted: id=0x0, method=, argCount=0 right before the fatal "system integrity has been compromised" abort. Also tried the Qwen3-specific single-phase online-softmax kernel — same failure. The generic processHeadsFlashAttention kernel (already used by Llama, and known-working on Metal) does not have this problem. Fixed by making both Qwen3FP16FFNLayers and Qwen3Q8_0FFNLayers select that kernel on Metal only (mirroring the existing isWarpShuffleSupported() backend-detection pattern), skipping the now-unneeded combine phase.

Verified directly on hardware (not just built): Qwen3-0.6B in F16 and Q8_0, standard and --with-prefill-decode, plus a Llama regression check — all run cleanly on both 5.2.0-jdk21-metal and 5.2.0-jdk25-metal, producing real inference output at ~15-17 tok/s with no crash.

@orionpapadakis orionpapadakis added bug Something isn't working CI Improvements related to CI automation 1.0.1 labels Jul 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

1.0.1 bug Something isn't working CI Improvements related to CI automation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant